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Editor’s Note 


A significant cybersecurity event has recently been discovered in which malicious actors gained access to the 
source code for the Orion monitoring and management software made by the company SolarWinds and inserted 
malware into that source code. This article contains brief perspectives from a few members of the /EEE Security & 


Privacy editorial board regarding that incident. 


A serious cybersecurity event 
was recently revealed: mali- 
cious actors had gained access to 
the source code for the SolarWinds 
Orion monitoring and manage- 
ment software. They inserted mal- 
ware into that source code so that, 
when the software was distributed 
to and deployed by SolarWinds 
customers as part of an update, the 
malicious software could be used 
to surveil customers who unknow- 
ingly installed the malware and 
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gain potentially arbitrary con- 
trol over the systems managed by 
Orion. Of course, such a level of 
control has given attackers oppor- 
tunities for further exploitation 
as well. 

At the time of this writing, it is 
reported by SolarWinds that the 
update containing the malware was 
installed by thousands of custom- 
ers, including numerous U.S. fed- 
eral agencies and businesses around 
the world. The cybersecurity com- 
pany FireEye was among the first 
reported to be actually compro- 
mised by the malware. 
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Numerous technical details on 
this incident appear online and in 
a variety of other publications as 
well as from SolarWinds itself.! 
At the same time, there remains 
a great deal to say about how we 
might best respond to and recover 
from the incident as well as how we 
might avoid similar such events in 
the future. 

The news of this event broke at 
a time that made it infeasible for 
IEEE Security & Privacy to write a 
full, detailed piece by the produc- 
tion deadline for this issue. How- 
ever, due to the magnitude of the 


March/April 2021 


J PERSPECTIVES 


incident, the Editorial Board still 
wanted to address it in some way 
without waiting two months for 
the next issue. Therefore, this issue 
contains two pieces related to the 
SolarWinds incident: This first 
article contains brief perspectives 
from some members of the IEEE 
Security & Privacy Editorial Board, 
including numerous questions 
asked by Editorial Board members 
and also some suggested solutions. 
The second is a companion “Point- 
Counterpoint” column article by 
Fabio Massacci and Trent Jaeger 
that digs specifically into the quan- 
daries and questions of software 
patching that relate to the Solar- 
Winds incident. Additional details 
will undoubtedly continue to sur- 
face, and IEEE Security & Privacy 
expects to cover this occurrence fur- 
ther and in greater detail in future 
issues as we continue to learn about 
this compromise and its effects. 

— Sean Peisert 


Editorial Board Members’ 
Perspectives 


SolarWinds and Market 
Incentives 


Bruce Schneier 
he penetration of government 
and corporate networks world- 
wide is the result of inadequate 
cyberdefenses across the board. 
The lessons are many, but I want to 
focus on one important one we’ve 
learned: the software that’s man- 
aging our critical networks isn’t 
secure, and that’s because the mar- 
ket doesn’t reward that security. 
SolarWinds is a perfect example. 
The company was the initial infection 
vector for much of the operation. Its 
trusted position inside so many critical 
networks made it a perfect target for 
a supply-chain attack, and its shoddy 
security practices made it an easy target. 
Why did SolarWinds have such 
bad security? The answer is because 
it was more profitable. The company 
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is owned by Thoma Bravo part- 
ners, a private-equity firm known 
for radical cost-cutting in the name 
of short-term profit. Under CEO 
Kevin Thompson, the company 
underspent on security even as it 
outsourced software development. 
The New York Times reports that the 
company’s cybersecurity advisor 
quit after his “basic recommenda- 
tions were ignored.” In a very real 
sense, SolarWinds profited because 
it secretly shifted a whole bunch of 
risk to its customers: the U.S. gov- 
ernment, IT companies, and others. 

This problem isn’t new, and, while 
it’s exacerbated by the private-equity 
funding model, it’s not unique to 
it. In general, the market doesn’t 
reward safety and security—espe- 
cially when the effects of ignor- 
ing those things are long term 
and diffuse. The market rewards 
short-term profits at the expense 
of safety and security. (Watch and 
see whether SolarWinds suffers any 
long-term effects from this hack, or 
whether Thoma Bravo's bet that it 
could profit by selling an insecure 
product was a good one.) 

The solution here is twofold. 
The first is to improve government 
software procurement. Software 
is now critical to national security. 
Any system of procuring that soft- 
ware needs to evaluate the security 
of the software and the security 
practices of the company, in detail, 
to ensure that they are sufficient to 
meet the security needs of the net- 
work they’re being installed in. If 
these evaluations are made public, 
along with the list of companies 
that meet them, all network buyers 
can benefit from them. It’s a win for 
everybody. 

But that isn’t enough; we need a 
second part. The only way to force 
companies to provide safety and secu- 
rity features for customers is through 
regulation. This is true whether we 
want seatbelts in our cars, basic food 
safety at our restaurants, pajamas that 
don’t catch on fire, or home routers 


that aren't vulnerable to cyberattack. 
The government needs to set mini- 
mum security standards for software 
that’s used in critical network applica- 
tions, just as it sets software standards 
for avionics. 

Without these two measures, 
it’s just too easy for companies to 
act like SolarWinds: save money by 
skimping on safety and security and 
hope for the best in the long term. 
That’s the rational thing for compa- 
nies to do in an unregulated market, 
and the only way to change that is 
to change the economic incentives. 


Hamed Okhravi 
T he SolarWinds hacks are not 
novel or unique. Software Tro- 
jans and supply-chain attacks have 
been understood in the community 
for many decades; as a case in point, 
even the logo of the IEEE Sympo- 
sium on Security and Privacy, one 
of the top-tier venues in computer 
security, is a Trojan horse! However, 
the SolarWinds hack highlights some 
of the troubling trends and impor- 
tant lessons that we, as a community, 
should pay attention to and work on 
resolving with better technologies as 
well as policies. 

First, for decades, one of the 
main paradigms in security has been 
“additive” security. We “add” tools, 
antiviruses, intrusion detection/ 
prevention systems, monitoring 
and management tools, and so 
on to make a system more secure. 
Every new tool, while it might 
reduce parts of the attack surface, 
also adds a new attack surface: vul- 
nerabilities in the tool itself become 
a new possible vector of compro- 
mise for the system. Numerous 
recent vulnerabilities discovered in 
security software further highlight 
this tradeoff.” 

This is not an easy problem to 
resolve. While some security soft- 
ware programs close many more 
holes than they open, others, by 
the virtue of their sheer size and 
complexity, can make the system 
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more vulnerable. We, as a com- 
munity, do not have a proper, 
quantitative understanding of this 
tradeoff, and, thus, decisions for 
deploying or not deploying a new 
tool are made without proper 
understanding of their implica- 
tions. This also highlights the need 
for another paradigm that requires 
more attention in the community: 
“reductive” security, or making a 
system more secure by removing its 
unnecessary services, libraries, fea- 
tures, and so on, thereby shrinking 
its attack surface. 

Second, software supply-chain 
attacks are as important—if not 
more important—than hardware 
supply-chain attacks. Because of 
the vast complexity of modern 
software and the comparatively 
low cost of introducing malicious 
logic into the code, this threat vec- 
tor can be expected to become 
more prominent over the next few 
years. The traditional security 
mechanisms that are developed to 
ward off malicious “outside” code, 
such as code signing, digital certifi- 
cates, secure update delivery mech- 
anisms, transport security, and so 
on, have little impact on this threat 
vector, as evidenced by the Solar- 
Winds hack. Deeper analysis tech- 
niques, code provenance tracking, 
and liability mechanisms as well 
as novel technologies and refined 
policies are necessary to mitigate 
such attacks. 

Finally, cobbling together soft- 
ware code and libraries from various 
and sometimes unknown sources may 
be a good way of developing tools 
quickly and cheaply; it is certainly 
not an effective way to develop tools 
securely. Large software projects 
often incorporate code and libraries 
from many sources with unknown 
provenance. At best, these code 
pieces can be buggy or faulty, and, at 
worst, they can contain malicious, 
implanted logic. Rapid and secure 
software development remains as 
tantalizing a goal as ever. 
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Fabio Massacci 

olarWinds offers network and 

database monitoring and log- 
ging services. SolarWinds, like all 
modern IT suppliers, grew out of 
mergers and acquisitions. A sim- 
ple investigation on the company’s 
FAQ page! shows that the “vulner- 
able platform” is, in reality, a patch- 
work of services: 18 (sub)products 
are, indeed, affected by the vulner- 
abilities (from Access Management, 
to Web Performance Monitor, to 
Log Analyzer—which was recently 
acquired); another 40+ (sub)prod- 
ucts are not affected. 

So, for example, Log Analyzer 
is affected, while Kiwi Log Ana- 
lyzer, Log & Event Manager, Log 
and Event Manager Workstation 
Edition, and Loggly are not. The 
Database Performance Analyzer 
Integration Module is affected, but 
the Database Performance Analyzer 
is not—or, more precisely, “we do 
not believe is affected.”! 

You have no chance as an end 
administrator to know what you are 
getting onto your systems. Figure 1 
illustrates the stark reality for mod- 
ern administrators. 

Of course, as all system-monitoring 
tools, SolarWinds systems are properly 
integrated with various mechanisms 
of authentication (such as Microsoft 


(a) 


Figure 1. 


(a) How we think the world is with good security gates and processes. 
(b) How the world really is. In the framework of a week, SolarWinds fetched a 
vulnerability (SUNBURST) to find out that another one was there (SUPERNOVA). 
(Source: Anna Formilan for the European Union AssureMOSS project. Used 

with permission.) 


Active Directories), possibly with 
single sign-on (SSO) (which is a 
good security practice, isn’t it? But 
I will return to that) at the final 
user sites. If one of these system 
tools from the patchwork is com- 
promised, the end user administra- 
tors are likely compromised. This is 
indeed quite clear from the instruc- 
tions in the Emergency Act 21-01.4 

What lessons do we learn from 
this story? 


« This is not about third-party soft- 
ware; it is about fourth- or eighth- 
or 16th-party software. We don’t 
know what we are installing, and 
even the people who sell it to us 
have no clear idea. 


All security procedures are design- 
ed for first-party software. (Just 
think about threat analysis and 
STRIDE, or quality gates, and so 
on.) We can claim that suppliers 
should stick to security proce- 
dures and so forth, but are we will- 
ing to pay the price? If nobody is 
liable for anything. ... 


Several security mechanisms (like 
SSO) that we conceived to be used 
with first-party products (which 
are secured as first-party products) 
can be damning in the new world 
order, as you SSO into something 
you have no idea about. 
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(b) 
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Terry Benzel 
T here has been a lot of impor- 
tant technical discussion to 
date, but I would also like to see 
some conversation about what the 
threat is, what the motivations are 
for exploiting the vulnerabilities, 
and how those fit into the larger 
social—political landscape. Given 
the extent of access gained, what is it 
being used for? Is it “simply” export- 
ing data and information? Are the 
accesses being used to change the 
information/platforms/integrity of 
data and processes? 

The wide variety of organizations 
that have been attacked makes it dif- 
ficult to attribute a single motivation. 
How do we close all of the doors that 
have been opened and understand 
what was done during the long time 
that access was available? 

Finally, we should note that this is 
not unique and not specific to adver- 
saries. This is just a highly visible 
move in a grand game that has been 
ongoing for a very long time. Under- 
standably, digging very deeply in this 
area will quickly become sensitive, 
but I do think that we owe the read- 
ers a bit of discussion about the big- 
ger picture. 


Carl Landwehr 

e all accept digitally signed 

software updates. Solar 
Winds evidently distributed val- 
idly signed updates that included 
malware, so customers who trusted 
them and installed the sabotaged 
updates became vulnerable. Perhaps 
the first time this kind of attack was 
written about was in Ken Thomp- 
son’s Turing Award lecture “Reflec- 
tions Trusting Trust” in 1984. What 
can be done about this problem 
from a technical standpoint? One 
possibility might be to develop 
tools that would help consumers 
of updates assess their security 
more deeply than merely check- 
ing that a digital signature is valid. 
The signature provides provenance 
assurance and nothing more, and if 
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the signing key has been compro- 
mised, even that is lost. 

What if the consumer could 
run checks on the signed update to 
check it for vulnerabilities? This is 
the approach taken by the Securely 
Taking on New Executable Software 
of Uncertain Provenance (STONE- 
SOUP) program at Intelligence 
Advanced Research Projects Activ- 
ity in 2009. Of course, this approach 
will never ensure there are no vul- 
nerabilities, but it could raise the bar 
substantially. Considerable technol- 
ogy has been developed by STONE- 
SOUP and subsequent programs, 
including DARPA’s Cyber Grand 
Challenge, that demonstrates the 
ability to perform significant auto- 
mated analyses on both source code 
and binaries to expose potential vul- 
nerabilities (and, in some cases, even 
to remove them). This technology is 
commercially available. 

Even better would be to put the 
responsibility on developers, who 
are in the best position to apply seri- 
ous effort to ensuring that their 
updates don’t contain malware. Why 
not establish best practices for the 
release of updates and make those 
developers who can't show they fol- 
lowed these practices liable for the 
malware they distribute? If we want 
to improve the resilience of our soft- 
ware systems, why don't we start by 
establishing liability for companies 
who distribute sabotaged updates? 
Nation-state attackers may still pene- 
trate defenses, and relief from liability 
should be available to companies that 
can show they followed the best prac- 
tices. Establishing liability requires 
policy action, but perhaps the severity 
of the Solar Winds compromise can 
provide sufficient political will to help 
the software industry begin to assume 
responsibility for its products. 


Mohammad Mannan 
ecurity products, like other soft- 
ware applications, come with 
numerous vulnerabilities and design 
flaws, and are exploited in cyber 


attacks every now and then. As of Jan- 
uary 26, 2021, a quick search at cve. 
mitre.org on SolarWinds/McAfee/ 
Symantec returns hundreds of 
Common Vulnerabilities and Expo- 
sures (CVE) entries. SolarWinds is 
making the headline now, but will 
possibly be forgotten sometime 
soon (recall the RSA hack from 
2011). I would argue that security 
products should be a distinct soft- 
ware category—perhaps be made to 
follow some standards. Other soft- 
ware categories provide direct and 
useful functionalities. In contrast, 
security products don’t enable any 
new functionality—designed only 
to secure other systems and applica- 
tions, and as such, must not create 
new security nightmares. 

Some suggestions along this line 
seem apropos: 


» Security products should do no 
harm. They should be designed 
with failure in mind—ie., they 
must fail gracefully, without 
reducing the security level of the 
protected system compared to 
the original state (without the 
product itself). This will require 
them to work with less privilege. 
To some extent, these products 
should be treated like medicine/ 
vaccine—the bottom line should 
be to avoid any harm. 


The operating system (OS) itself 
should take over more responsi- 
bilities gradually for tasks requir- 
ing system-level access. (The OS 
is the necessary evil that we need 
to keep secure anyway.) 

Security products need to accept 
liabilities if they downgrade the 
target system’s security in any 


way—this can force them to 
become a security company (ie., 
adopt the best security practices, 
including in development/deploy- 
ment/management phases) as 
opposed to yet another software 
company that happens to make 
security products Instead of 
requiring an abuse to happen to 
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trigger penalties, another avenue 
could be to incur a certain amount 
of fine for each new CVE, and 
after a certain threshold, the prod- 
uct or company may lose their 
“security” label. 


Of course, these suggestions 
here are incomplete and need more 
serious consideration. However, 
if no concrete measures are taken 
soon, we should not be surprised 
when (not if) the next SolarWinds 
like incident happens. 


Jelena Mirkovic 

tis important to note that this was 

a well-planned, well-executed 
attack. For example, the changes in 
code were done at the source code 
level over months and possibly by 
an insider.> The changes were sub- 
tle, using coding styles and nam- 
ing conventions that were already 
in the code. From that point on, 
the code was tested, signed, and 
released through the SolarWinds 
platform. After infection, the code 
was dormant for two weeks. After 
that, it mimicked another known 
protocol to exfiltrate credentials. 
These credentials were used to 
access Security Assertion Markup 
Language token authority and craft 
new tokens. These tokens were then 
used to access confidential data. 

In my opinion, there were two 
weak links that the attack exploited: 
insiders in the companies producing 
software/hardware and _ installing 
monitoring software that precedes 
encryption (otherwise, credentials 
could not be stolen). The first is 
impossible to eliminate. The sec- 
ond—perhaps, but there will always 
be tension between the desire to 
monitor everything and protecting 
privacy (in this case, the privacy of 
credentials that were later used to 
compromise the security of data). 

I think the better question is 
how one could detect when such 
high-level compromises occur. Once 
stolen, data have to be exfiltrated 
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somehow. In this specific case, new 
Internet destinations were con- 
tacted by compromised machines, 
and data were exfiltrated. This could 
potentially be detected, although 
command and control servers were 
hosted on public clouds (Microsoft 
and Amazon), which may not have 
looked suspicious. Therefore, the 
third weak link is our reliance on 
clouds and allowing organizations 
with high confidentiality require- 
ments to communicate with pub- 
lic clouds. 


Atul Prakash 
T he issues surrounding the 
SolarWinds incident raise 
important questions. First and fore- 
most, this is clearly a supply-chain 
attack. I think that a number of my 
questions are around potential priv- 
ileges, both in this incident specifi- 
cally and in software supply chains 
more generally. In this incident, were 
the compromised SolarWinds serv- 
ers more overprivileged than neces- 
sary to function properly? Toward 
answering this, what capability did 
SolarWinds provide, and how is 
it typically deployed? Why did its 
compromise then lead to the com- 
promise of internal email accounts, 
even assuming that the SolarWinds 
server was fully compromised and 
gave root access? Does the server 
act as a man in the middle, or is it 
also installed at end servers, such 
as mail or authentication servers? 
If it is a man in the middle, why did 
end-to-end authentication fail to 
protect mail accounts? How many 
other such third-party services that 
could allow deeper compromise of 
other services are out there? 


James Bret Michael 

amage assessment will likely 

take months to accomplish, 
given the reach of SolarWinds 
(ie, the number of customers and, 
through transitivity, other parties 
affected) and the large number of 
organizations (e.g., the U.S. Federal 


Bureau of Investigation, the United 
States Cyber Command, and secu- 
rity firms) involved in the investiga- 
tions of SolarWinds. 

Cyber hygiene still seems to be a 
stumbling block to thwarting the suc- 
cessful use of low-hanging exploit- 
able vulnerabilities. Of the 300,000 
customers (granted, not all use the 
Orion software or were still licensed 
to do so), a relatively small number 
of customers actually uploaded the 
updates. Preventing another Solar- 
Winds debacle is a technical (e.g, 
eliminating unnecessary system 
complexity to make it feasible to 
assess the trustworthiness of the soft- 
ware/firmware updates and the orga- 
nizations providing those updates) 
and human problem (eg,, getting 
the bureaucracies in organizations to 
address supply-chain issues). 

There doesn’t seem like there 
is much we can do to solve the 
supply-chain problems. Software, hard- 
ware, and co-design are global. Lots of 
systems and apps comprise compo- 
nents sourced from multiple suppli- 
ers. Governments do not control the 
supply chain, just as they do not con- 
trol innovation or how products and 
services are used. Governments have 
been somewhat ineffectual in handling 
supply-chain issues. They complain 
a lot but don’t seem to have workable 
solutions or good leadership in the 
supply-chain arena. 

SolarWinds appears to have 
involved one or more insiders and 
external actors, making it challenging 
to perform a full damage assessment 
and nearly impossible to detect adul- 
terated software components. Like I 
commented before, I don’t see any- 
thing new here in SolarWinds. It is just 
that a lot of people are embarrassed 
(including top cybersecurity firms), 
and the media is having a field day. 
Why weren't the more sophisticated 
users of the software monitoring their 
own systems more closely? 

It will be interesting to see the 
forthcoming attempts to apply 
artificial intelligence to detect the 


11 


J PERSPECTIVES 


12 


types of reconnaissance, command 
and control, and other aspects of 
penetration that were employed by 
SolarWinds. However, the first step 
should be to use a bit of common 
sense to think through the problem 
before trying to apply AI. 

SolarWinds takes advantage of 
vulnerabilities in widely used plat- 
forms (.net) and protocols (eg, 
Domain Name System). Can they 
be fixed? What does it take to get 
them fixed or replaced? 

Initial findings indicate that adv- 
anced tactics, techniques, and pro- 
cedures (TTPs) were employed by 
SolarWinds. The cyberdefense 
teams are always trying to catch up 
with the latest TTPs and new imple- 
mentations of well-known TTPs 
used by the attackers. Addressing 
TTPs requires better communica- 
tions between the offensive and 
defensive sides of the house. 

Overreaction by governments to 
SolarWinds could have a profoundly 
negative effect on global innovation. 
Then again, governments are not in 
the driver’s seat here. The private sector 
will continue to innovate. 

The way forward may be to take a 
fundamentally new approach to secu- 
rity in the near- (e.g, cloud hypervisors 
pushing security rather than individual 
customers patching—or not), mid- 
(e.g., zero-trust, or something like it), 
and long-term (eg, quantum-based 
approaches). The other part of the 
equation that needs to be addressed 
is motivation—getting the market to 
include security as a requirement. = 


Disclaimer 

The views and conclusions contained 
herein are those of the authors and 
should not be interpreted as neces- 
sarily representing the official policies 
or endorsements, either expressed or 
implied, of the authors’ employers. 
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